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ABSTRACT * 

Evaluation is important to publ i c education and 
training programs if the United States is to be competitive in a 
global economy. Six major factors have had an impact on evaluation 
efforts: decreased financial resources, increased public 
dissatisfaction, changed management approaches, enhanced evaluation 
methods and procedures , redesigned organizations , and increased 
privatization efforts. Today, three strategies are being used for 
evaluating educational programs , and they are likely to continue to 
be used into the next century. These strategies are outcome 
assessment, performance assessment and measurement, and value-added 
assessment. Ten recommendations are offered to improve evaluation 
efforts: (1) build a commitment to conducting high-quality 
evaluations; (2) develop individuals* evaluation capacity; (3) foster 
a clearer understanding of mission, customers and their needs, and 
desired results; (4) find a closer link between performance measures 
and the outcomes that they asses; (5) focus programs on a limited 
number of outcomes for which they will be accountable; (6) programs 
should not avoid having outcomes that are difficult to measure; (7) 
strengthen the systems and processes for generating and using 
f inane ial and program information in conducting program evaluat i on ; 

(8) use greater innovation and creativity in developing evaluation 
and accountability techniques; (9) develop more efficient and 
cost-effective evaluation and assessment techniques; and (10) keep 
politics out of the process, (Contains 24 references,) (KC) 
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Foreword 

Achieving the rank of Professor in a major research university signifies the pro- 
fessional maturity of the scholar and gives the individual full rights as a member of 
the academy. Promotion to Professor is a noteworthy event. 

Therefore, when a person is named Professor, that accomplishment should be 
marked by offering a significant work ol cholarship as a contribution to the disci- 
pline. The Department of Agricultural Education at The Ohio State University es- 
tablished the "'Professional Inaugural Lecture Series" in 1988 to provide a forum in 
which such a scholarly work can be presented. This series also is designed to pro- 
vide signal recognition to the new Professor. 

This paper, prepared by N. L. McCaslin, is the second in the series. 

— R. Kirby Barrick 
Chair 
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Developing a strong internationally competitive economy depends, in part, on 
preparing workers to compete in the work force. Our country has invested consid- 
erable resources in public education and training programs designed to assist indi- 
viduals with the process of work force preparation. These formal and informal pro- 
grams are offered through agencies such as secondary schools, community and 
technical colleges, and colleges and un'versities. They range from short-range job 
upgrading activities to long-range gracuate education and result in certificates, 
diplomas, degrees, and licenses designed to verify competence. Familiar examples 
of these programs include vocational education programs offered by secondary 
schools and community and technical colleges, degree programs offered by The 
Ohio State University on its various campuses, programs conducted by the Ohio 
State University Extension, and Job Training and Partnership Act (JTPA) training 
programs offered for at-risk individuals or displaced workers. Education and train- 
ing programs also are offered in the private sector through groups such as propri- 
etary schools, businesses, and industries. Examples include certificate and degree 
programs offered by private schools and colleges as well as training, upgrading, 
and retraining activities offered by business and industry. 

As we think about the future of these programs and how they might be im- 
proved, I am reminded that in less than five years, a new millennium and a new 
century will arrive. Cetron and Davies (1989) indicated that ". . . the end of one 
century and the beginning of another is a time when creativity flows with special 
freedom" (p. 317). They argue that, for some unknown reason, this end-of-century 
phenomenon provides the momentum or spontaneity for developments that no one 
could have predicted. 

Today, I would like to share some thoughts about ev^aluation efforts for public 
education and training programs as v/e approach the 21st century. First, I will iden- 
tify six major factors that have had an impact on ev'aluation efforts. Next, I will dis- 
cuss three evaluation strategies that are currently being advocated and likely to 
continue through the next decade. I will conclude with some recommendations for 
improving evaluation efforts related to public education and training programs. 

Factors That Have Had an Impact on Evaluation 

Never in the history of the United States has evaluation of educational pro- 
grams received more atten.ion than today. Since the release of the National Com- 
mission on Excellence in Education's (1983) A Nation At Risk: The Imperative for Edii- 
catw)ial Reform, there has been a profusion of reports critical of the quality of educa- 
tion in this country. These reports include: The Forgotten Half: Non College Youth in 
Anierica (The William T. Grant Foundation Commission on Work, Family and Citi- 
zenship, 1988); America’s Choice: HirIi Skills or Low WaRes! (Commission on Skills of 
the American Workforce, 1990); America 2()0(): An Ediication StratcR ./ (U. S. Depart- 
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V)nn (qJ 'I" of^dioub: A SCANS Report for America 

Co»» S^fsnpr Achieving Necessary Skills, 1991); Education 

on Education Indicators, 1991); and Learning a Livhw: A 

Ski lTwm tI (Secretary's Commission on Achieving Necessary- 

Skills, 1992). These studies have called for major changes in the way we plan de- 

iver, and evaluate our educational programs. They also stressed the gap^between 
and aZst ^““m^r^r youth 

andtinitp.^^^^^ 

creased public ciissatisfaction, changed management approaches, enhanced 

;:tSa”on:tr1r "" 

Decreased Financial Resources 

levels^ At one Hme competition for funding at the local, state, and federal 

hWp education was a major user of tax funds. Today, competition for 

these scarce resources comes from agencies such as law enforcement, sodal ser- 
vices, transportation, health, employment services and the list goes on. 

Pressures are increasing to reduce the federal deficit. These pressures have re- 
sulted from ' the Republican Congress's election pledge and the Democratic 

f9TO p » ThruTrlr I 'i'”'"'’"' (Jennings & Stark, 

defiVit li !i f ■ T' Amounting Office (1993) reported that moving from ^ 

men! ri Surplus IS essential to improve national savings, private invest- 
ed" ' report went on to say [hat redu^the 

ederal budget deficit is a vital action that needs to be taken if future workers are to 

def^ taken to reduce the 
deficit It will increase to 20 percent of gross national product (GNP) principally due 

o rising hea th retirement, and the associated interest costs. BecauL of these L 
b^n eSatS'"^' Programs have received severe budget cuts or 

Increased Public Dissatisfaction 

Second the public and policy makers often are unclear about what they are 
pttmg for their money and demanding stricter accountability for the resources 
xpen e . le public has lost faith in many organizations and institutions ques 

e needs of society. This has resulted m numerous calls to "do more with less" and 
no mom business as usual." Schools and olher educational agencies are behig 
sked to strive for excellence and produce individuals who wfll become woild-chss 
workers at lower costs and with greater effectiveness. 

Changed Management Approaches 

mov™'from management and accountability has 

rnmpc fh 1 J-mphasis on inputs and proces.ses to a greater focus on the out- 
comes that are being achieved. All too often public education and training nro- 

fnTid-iv' kigh cost, and low produ^ctivity. 

In today s environment the.se characteristics cannot be tolerated and the federal 

gc vernment has enacted legislation to mandate more accountability. Performance 
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•''nd mo.isiiro? \\’ero first adonfc'd as an instrunic'nt of nni’'"':'' i' !''nmn 
resource policy with the passage of the Job '1 raining Partnership Act (j 1 PA) of 1982. 
The measures developed for these programs included: placement and retention in 
unsubsidized employment, earnings, and reduction in public assistance. The Carl 
D. Perkins Vocational and Applied Technology Education Act of 1990 required 
states to develop core standards and measures of performance for secondary and 
postsecondary vocational education programs. These performance measures were 
to include learning and competency gains in basic and :> -n advanced academic 
skills and at least one or more measure(s) of performance including competency 
attainment, job or work skill attainment or advancement; retention in school or 
completion of secondary school; and placement into additional training or educa- 
tion, military service or employment. The Government Performance and Results 
Act (GPRA) requires each agency to submit a strategic plan to the Office of Man- 
agement and Budget (OMB) and Congress by September 30, 1997, covering pro- 
gram activities for at least five years; an annual program performance plan to OMB 
beginning in FY 1999; and an annual program performance report to the President 
and Congress beginning in 2000 that covers the previous fiscal year. 

Enhanced Evaluation Methods and Procedures 

Fourth, there have been increased efforts to improve program evaluation. Over 
the past two decades, a total of 16 professional education associations have worked 
together to develop a set of 30 standards that identify principles that, when ad- 
dressed, should result in improved evaluations of educational programs, projects, 
and materials (The Joint Committee on Standards for Educational Evaluation, 

1994). The standards were first published in 1981 and revised in 1994. These stan- 
dards were developed around four attributes: utility, feasibility, propriety, and accu- 
racy. The utility standards were designed to ensure that evaluations would serv'e 
the information needs of intended users. The feasibility standards were to help 
evaluations to be realistic, prudent, diplomatic and frugal. The propriety standards 
were designed to see that evaluations were conducted legally, ethically, and with 
due regard for the welfare of those involved in the evaluation and those affected by 
its results. Finally, the accuracy standards were to ensure that an evaluation will 
reveal and coiwey technically adequate information about the features that deter- 
mine worth or merit of the program being evaluated. Additionally, groups such as 
the Nalional Science Foundation (Frechtling, 1995) have sponsored papers and con- 
ferences designed to propose new and innovative evaluation methodologies that 
could help improve both educational programs and evaluations. These new initia- 
tives should be both cost-effective and feasible. Also, greater attention should be 
given to breaking down the barriers between quantitative and qualitative method- 
ologies. 

Redesigned Organizations 

The world has changed dramatically in the past few years and so have the 
needs and expectations of the people. The United States has quickly moved from a 
producer to a consumer economy. Others have characterized this shift as moving 
from the industrial to the information age. In moving to this new type of environ- 
ment, many organizations finally are asking their clients and customers what their 
needs are rather than telling them. Consequently, these organizations are undergo- 
ing a dramatic redesign of their operations. To some it is viewed as "reinventing," 
to others it is "downsizing," and yet to others it is viewed as "rightsizing." Much of 
this redesign is the result of feeling that organizations have become too large, insen- 




imro'^non'^ivo, nnd co=^l!y. Tor cxnmj'ile, The Ohio State Univc'v i'" ' id('nt 

li. Gordon Gee, in his speech at the Project Reinvent Symposium (October 11, 1994), 
said that . . Ohio State cannot be all things to all people." He went on to say that 
"... 1 often feel like Noah, with two of everything around this place. Not any more." 

Typically these redesign efforts have taken radical thinking about "what an 
organization should do" and "how it should be done." A former colleague, now 
deceased, used to remind me "if we keep on doing what we're doing, we're going 
to keep on getting what we've got." We need more flexible, entrepreneurial, decen- 
tralized organizations to meet the societal needs of the 21st century. 

Increased Privatization Efforts 

Another major force influencing evaluation is the attempt to privatize many 
public programs such as education and training. This force is based on the view 
that competition will increase quality and productivity and decrease cost. It is fur- 
ther supported by the opinion that the private sector can do it better, faster, and at 
less cost. Much of this view is based on the premise that public programs are too 
antiquated, wasteful, duplicative, unfocused and bureaucratic. A recent study by 
the U.S. General Accounting Office (1994) found that there were 154 employment 
and training programs operated by the Federal Government. Efforts to privatize 
public efforts have been seen in the U.S. Postal Service and similar efforts are being 
considered for the National Aeronautics and Space Administration. 

Evaluation. Strategies 

As can be seen from the previous section, educational agencies of all types are 
under increasing pressures to become more accountable. This accountability is be- 
ing required in the form of programs that better meet the needs of its constituency, 
have more effective educational processes, and result in higher levels of achieve- 
ment and competence. This section will address three specific strategies currently 
being used for evaluating educational programs: outcome assessment, performance 
assessment and measurement, and value-added assessment. 

Outcome Assessment 

Outcome assessment refers to "evaluative processes that determine the results 
of education at the institutional, program, and student level" (Bragg, 1992, p. iv). At 
the institutional level, outcomes assessment is often viewed as part of an institu- 
tional effectiveness evaluation (Alfred & Kreider, 1991; Nichols, 1991; Seybert, 

1990). As such, the results of outcome assessments often impact areas such as insti- 
tutional decision making, strategic and tactical planning, organizational adminis- 
tration, staff development, and curriculum development. Examples of outcomes 
assessments at the institutional level include graduation or completion rates, con- 
tinuing education rates, client satisfaction with services received, usefulness of 
management information systems, impact of collaborative activities, and results of 
efforts to redesign or "reinvent" organizations. 

Program outcomes, sometimes referred to as functional-area outcomes (Alfred 
& Kreider, 1991), provide an indication of the success of an organized sequence of 
courses and services. Examples of program outcomes include placement rates, stu- 
dents' satisfaction with their program of study or major, employer satisfaction with 
program graduates or completers, and number of student hours generated. 
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Student outcomes are those nssociated with changes tliat ornir in .■-'n *onfs c^r 
clients of an educational experience. These outcomes include those such as the 
knowledge of a subject, abilities or skills learned, attitudes developed, and higher- 
order thinking or problem solving skills. 

Performance Assessment and Measurement 

Performance assessment has been defined as "the process of estimating the 
value of or describing the performance attributes of or products created by an indi- 
vidual or a group" (Wheeler & Haertel, 1993, p. 1). Other terms that are often used 
to describe performance assessment include alternative assessment, authentic as- 
sessment and direct assessment. Mabry (1992) distinguished among these terms as 
follows: 

I Performance assessment signals that something students actually do is being 
evaluated. 

I Alternative assessment signals that evaluation is based on something other than 
standardized testing or similar formats. 

I Authentic assessment signals that assessment tasks presented to students elicit 
worthwhile responses reflective of real or real-world competence. 

I Direct assessment signals that the evaluation focuses on what we want to mea- 
sure rather than a proxy for it (p. 110). 

Performance measurement has been defined as "a process by which a program 
objectively measures how it is accomplishing its mission through the delivery of 
products, services, or processes" (Financial Management Service, 1993). Perfor- 
mance measurement consists of five phases: strategic planning, creating indicators, 
developing a data measurement system, refining performance measures, and inte- 
grating with management processes. The Government Performance and Results Act 
of 1993 (GPRA) requires that federally funded agencies develop and use perfor- 
mance measurement in implementing an accountability system. The use of perfor- 
mance measurement, according to GPRA, includes setting goals and objectives and 
then measuring progress toward their achievement. Currently, 71 pilots are under 
way across most federal agencies and range from individual programs to entire 
agencies. The Social Security Administration, the Internal Revenue Service, and the 
Defense Logistics Agency are examples of entire agencies that are piloting GPRA's 
performance planning and reporting requirements. Within the Department of Agri- 
culture, the Cooperative State Research, Education and Extension Service (CSREES) 
is implementing the provisions of the GPRA by grouping their 55 budget lines into 
the following program activity areas: 

I Agriculture 

I Natural Resources and Environmental Management 
I Nutrition, Diet and Health 

I Community Resources, Economic Development, and Family Resource Manage- 
ment 

I Family, Children and Youth Development 
I Research and Education Capacity Building 
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Vahie-Added Assessment 

Value-added assessment has been defined as the positive differences that an 
educational experience makes in a student's knowledge, attitudes and skills 
(Northeast Missouri State University, 1984). This approach to evaluation and ac- 
countability stresses the use of measurement about the attainment of outcomes and 
using instrumentation such as standardized tests, performance samples, essays, 
departmental exams, interviews and surveys to monitor student progress. Using 
these types of instruments, measurement is taken when a student first enters an 
institution, major, program, or course. These initial measurements can be very help- 
ful in diagnosing students' needs, enabling students to understand the rationale for 
a curriculum, assessing the amount of improvement students must make, and pro- 
viding a baseline for the comparison of later performance. Measurements are again 
taken as students progress through or leave an institution, major, program or 
course. The information collected as individuals pass through an institution, major, 
program, or course can provide important feedback on their progress. Information 
collected at the end of an educational program may also be used to certify compe- 
tence. The value-added approach to evaluation and accountability can also provide 
information regarding the influence of an organization's actions upon desired out- 
comes. The comparison of later scores with those obtained at entry is what estab- 
lishes it as a value-added approach. 

Well-managed organizations measure their performance and then they compare 
themselves with the best organizations of their type in benchmarking efforts. 
Benchmarking is the continuous process of measuring one's own products, ser- 
vices, and practices against the world's toughest competitor — or those recognized 
as leaders in the particular aspect you're studying — to identify areas for improve- 
ment (Ford, 1993). What would you find if you used benchmarking to examine 
training efficiency using the training costs per student hour; training results using 
the average percent gain in learning per course; training results using average per- 
cent of improvement in on-the-job performance after training; or training activity 
using average training hours per employee? 

Suggestions for Improving Evaluation Efforts 

As we approach a new century and millennium, it is appropriate for us to con- 
sider what might be done to offer the best possible programs, be mo. e accountable, 
and help build the work force for the 21st century. Listed below are ten recommen- 
dations for improving our evaluation efforts. 

1. We need to build commitment to conducting high quality evaluations and 
then use this information for accountability and program improvement efforts. Ef- 
forts must be undertaken to recognize and communicate the benefits of evaluation. 
Better understanding of and closer working relationships with planning and evalu- 
ation need to be established. All too often evaluation is only thought of as a pro- 
gram or activity is ending. Program operators and educators are often wary of 
evaluation and accountability efforts. They fear that they may be held responsible 
for outcomes over which they have little or no control. People often view evalua- 
tion positively only when someone else or another person's program is being evalu- 
ated. Program administrators and policy makers sometimes fail to follow through 
with evaluation recommendations to alle’date problems or reward individuals and 
programs that achieve at high levels. 



nrr fo bn\’(' I'h' \ r>iv^ r''li>")hlo informnlion n'qnin’ci tn ip-inr".’C' edu- 
cation and training programs we need to develop individuals' evaluation capac- 
ity — knowledge, skills, and attitudes. Organizations that fail to improve themselves 
will not remain viable. Individuals must develop the ability to use multi-methods 
in conducting evaluations. These multi-methods should include both quantitative 
and qualitative techniques. Individuals need to know how to involve stakeholders 
in identifying important evaluation cjuestions. Greater skill is also needed in learn- 
ing to state outcomes, goals, and objectives with more precision and then to select 
appropriate measures. 

3. A clearer understanding of mission, customers and their needs, and desired 
results need to be fostered. Mission statements indicate why an organization exists 
in results-oriented terms. Organizations without clear formal mission statements 
tend to lack focus and direction. As Socrates said, "For a man [sic] without an in- 
tended port, no wind is favorable." Mission statements also need to be clearly un- 
derstood by all. 

Once the mission statement is developed, the clients and customers need to be 
contacted to find out their needs. Organizations often think that only they know 
what is best for their clients and customers. It is usually quite difficult for organiza- 
tions to change and pressures generally have to come from the outside. The experi- 
ences of the American automobile industry in the 70s and early 80s showed how 
easy it is to misjudge customer needs. We need to listen and respond to our clients 
and customers and then keep checking back with them to see how well we're doing 
in meeting their needs. 

4. A closer link is needed between performance nreasures and the outcomes th.at 
they assess. We need to measure what matters. In this way we can find out the de- 
gree to which strategic goals are being met. It also helps to gauge the impact a 
program's products or services are having. In this process, organizations will need 
to address multiple priorities — customer satisfaction, standards of quality, and cost 
limitations. Also, an organization should continuously compare its results with 
those obtained by the top organizations of their type. In reaching the goal of de- 
creasing costs, an organization must also be cognizant of the possibility of reducing 
customer satisfaction. 

5. Programs and activities should focus on a limited number of outcomes for 
which they will be accountable. A program cannot be "all things to all people." By 
limiting the number of outcomes, greater clarity can be achieved regarding the role 
and function of a program. Too many outcomes tend to confuse the target audience 
and the public and policy makers who are responsible for providing the funds to 
operate them. The program outcomes should be selected based on broad based 
input from program stakeholders. However, program operators also need to recog- 
nize that multiple stakeholders can make it difficult to reach consensus on the most 
important outcomes. 

6. Programs should not avoid having outcomes that are difficult to measure. Mea- 
suring the outcomes of a research program whose results may not be evident for 20 
years still should be addressed. Similarly, the investments in people may not hav'e 
immediate payoff. However, simply because something is difficult, or takes a long 
time to measure, should not serve as a "copout" for failing to address the outcome. 
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